Skip to content

feat(viewer,webview): embed QtMultimedia in AnthiasWebview, eliminate two-process DRM contention + Pi 4 drops#2905

Open
vpetersson wants to merge 10 commits into
perf/pi-vo-gpu-drmfrom
feat/embed-libmpv-webview
Open

feat(viewer,webview): embed QtMultimedia in AnthiasWebview, eliminate two-process DRM contention + Pi 4 drops#2905
vpetersson wants to merge 10 commits into
perf/pi-vo-gpu-drmfrom
feat/embed-libmpv-webview

Conversation

@vpetersson
Copy link
Copy Markdown
Contributor

@vpetersson vpetersson commented May 15, 2026

Issues Fixed

Closes #2904. Stacked on top of #2885 — base perf/pi-vo-gpu-drm. Merge order: 2885 first.

Description

The viewer's Qt6 video path was two processes contending for one framebuffer: AnthiasWebview (Qt) holding the surface, and an external mpv subprocess painting video via --vo=drm (linuxfb) or --vo=gpu --gpu-context=wayland (cage). PR #2885 documented 600-2800 vo drops per 60 s clip on Pi 4 even with HW decode engaged.

This PR moves video playback inside AnthiasWebview's Qt process via QtMultimedia (QMediaPlayer + QVideoWidget). Single Qt process, single GL/Wayland surface, no more mpv subprocess.

Implementation arc

I tried two embedding approaches; the second is what's shipping:

  1. libmpv via mpv_render_context — engaged HW decode correctly on every board (verbose mpv logs confirm drm-copy engaged on rpi-hevc-dec via /dev/video19, v4l2m2m-copy on bcm2835-codec via /dev/video10) but Pi 4 frame drops stayed at 600-2800/60 s. Root cause: libmpv-render uploads decoded frames to a GL texture, then QOpenGLWidget blits that into an offscreen FBO, then Qt's compositor blits the FBO into the eglfs window — three GPU passes per frame on a V3D 6.0 that can't sustain that at 60 fps. A QOpenGLWindow workaround to bypass the FBO crashed because eglfs is single-native-window-per-process. Branch history preserves the experiment; the final tree replaced it.

  2. QtMultimedia QMediaPlayer + QVideoWidget — ships. QVideoWidget paints inside MainWindow's existing eglfs native window (no second QWindow), Qt 6.8's ffmpeg-backed multimedia pipeline routes decoded frames through QVideoSink without the FBO indirection. The +rpt1 ffmpeg (already pinned in feat(viewer,server): per-board HW decode dispatch + codec gate on upload #2885's _rpt1-ffmpeg-pin.j2) carries --enable-v4l2-request --enable-v4l2-m2m, so the existing Pi-family hardware decoders (rpi-hevc-dec, bcm2835-codec, Hantro G2 on Pi 5, rkvdec on Rock Pi 4) all engage automatically via libavcodec without any per-codec dispatch from the application.

Real-device validation (BBB pack)

All three reachable Qt6 boards measured against the same 8-clip BBB pack (1080p / 4K, 30 / 60 fps, in H.264 and HEVC, each clip ~60 s @ ~8 Mbps). Drop counts come from the in-process counter in VideoView::onVideoFrameDelivered() (frames the QVideoSink received) compared to container_fps × position_s, written to /data/.anthias/mpv-stats.log per play (filename held for compat with the test-bed grep workflow). The numbers below are the median across n full-clip plays in a ≥1200 s observation window (Pi 5 ran 5 cycles; Pi 4 / x86 ran 2-3 cycles given the 8-clip rotation).

Asset x86 Pi 4 Pi 5 libmpv-era Pi 4 baseline
BBB 1080p30 H264 2 / 1798 (n=4) 3 / 1799 (n=3) — (HEVC-only gate) 600-1500
BBB 1080p30 HEVC 2 / 1799 (n=4) 2 / 1799 (n=3) 2 / 1799 (n=9) 562
BBB 1080p60 H264 0 / 3597 (n=5) 0 / 3594 (n=3) * 2973
BBB 1080p60 HEVC 3 / 3601 (n=4) 3 / 3601 (n=3) 2 / 3600 (n=10) 2402
BBB 4K30 H264 1 / 1798 (n=5) * 124 / 1464 (n=3) * 883
BBB 4K30 HEVC 1 / 1797 (n=4) 2 / 1798 (n=3) 1 / 1797 (n=9) 1183
BBB 4K60 H264 1 / 3597 (n=4) * 242 / 1713 (n=3) * 951
BBB 4K60 HEVC 3 / 3601 (n=4) 2 / 3598 (n=4) 2 / 3600 (n=9)

Format: dropped / expected frames (n = full-clip plays). * = derived from the viewer's STOP log line when asset_loop preempted the clip at its declared duration before QMediaPlayer::EndOfMedia could fire (expected = position_ms × container_fps).

The cells doing the heavy lifting for #2904:

  • Pi 4 — BBB 1080p60 H.264 dropped from 2973 frames/min (libmpv via subprocess and via in-process render context, both ≥ 2400) to 0. The stateful v4l2m2m-copy decoder is at real-time, and removing the QOpenGLWidget FBO indirection let V3D keep up with the present pipeline.
  • Pi 4 — every HEVC row is 2-3 drops per minute, regardless of resolution. rpi-hevc-dec (/dev/video19) drives it via v4l2_request from the +rpt1 libavcodec, and QtMultimedia's ffmpeg backend routes the decoded surfaces straight to QVideoSink.
  • Pi 5 — all four HEVC clips clear single-digit drops across 9-10 plays, including 4K60. Hantro G2 + Wayland (cage) is unchanged in this PR; the win is no longer losing the first 60 s after a viewer restart (loadImage('null') regression — see below).
  • x86 is uniformly clean.

Pi 4's 4K H.264 rows (124 and 242 drops/min) are the only outliers. Stream position at the 60 s wall-clock mark only reached 48.8 s (4K30) / 28.5 s (4K60) — that's the bcm2835-codec H.264 path running ≈ 80 % / ≈ 47 % of real time, not a presentation-side drop. This is a hardware limitation that pre-existed PR #2885; _HW_DECODE_VIDEO_CODECS['pi4-64'] would still accept these uploads, but the rest of the matrix shows the gate's intent (H.264 ≤ 1080p, HEVC at any resolution) is already what the hardware can sustain. Out of scope for #2904.

Rock Pi 4 validation is deferred — SSH banner exchange has been timing out on the testbed since the second 4K HEVC ingest stalled there earlier today. The arm64 image (69bb873-arm64) carries the same VideoView code path the other three boards just validated. Tracking the re-test separately.

Architecture changes

  • src/anthias_webview/VideoView is now a QWidget hosting QMediaPlayer + QVideoWidget + QAudioOutput. MainWindow exposes the same D-Bus surface (playVideo / stopVideo / videoEnded) as the libmpv variant, so the Python contract is untouched.
  • src/anthias_viewer/media_player.py — option dict shrinks from {hwdec, video-sync, vd-lavc-threads, audio-device, video-rotate} to just {audio-device, video-rotate}. _PI_HWDEC_BY_CODEC, _pi_hwdec_for_uri, and _probe_video_codec (ffprobe subprocess) are gone — libavcodec's auto-selection covers the per-board decoder dispatch.
  • docker/Dockerfile.viewer.j2 — drops libmpv2, adds libqt6multimedia6, libqt6multimediawidgets6, qt6-multimedia-dev. Pi 4 switches QT_QPA_PLATFORM from linuxfb to eglfs (libmpv-render needed a GL context; QtMultimedia keeps benefitting from one). The eglfs KMS config pinning 1920x1080 is shipped at docker/eglfs-kms-pi4.json.
  • tests/test_media_player.py — argv assertions become D-Bus options-dict assertions; per-codec dispatch tests dropped; rotation / audio / proxy / VLC-fallback tests preserved. 63 cases green.
  • src/anthias_webview/tests/ (new) — QtTest unit suite for VideoView (constructor builds player, stop idempotent, play with empty/unknown audio device, video-rotate passthrough). bin/test_webview_cpp.sh builds + runs them under QT_QPA_PLATFORM=offscreen. CI integration is a follow-up.

Key fixes during validation

  • view_image('null') was stopping the freshly-started video. src/anthias_viewer/__init__.py:495 calls view_image('null') AFTER media_player.play() to clear any leftover image/web background. My initial View::loadImage always called hideVideoSurface(), which stopped the QMediaPlayer 66 ms after PLAYING fired. On Pi 4 this was invisible (decoder past init); on Pi 5 it killed the Hantro G2 mid-CMA-allocation for 4K60 HEVC — 0 frames in 60 s. Fix: skip hideVideoSurface for the 'null' sentinel (commit).
  • QUrl(uri) with scheme-less local paths refused to load. QMediaPlayer silently rejected /data/anthias_assets/abc.mp4 because the QUrl had no scheme. Use QUrl::fromLocalFile for any path starting with /.
  • GLib.Variant wrap for D-Bus a{sv} — pydbus refuses to auto-coerce str to GLib.Variant when the slot is declared with QVariantMap. Wrap each option value via _marshal_dbus_options (regression test guards it).

Notes for upgrade

Pre-PR-#2885 testbed images carry QT_QPA_PLATFORM=linuxfb hardcoded in the device's deployed docker-compose.yml. The current docker-compose.yml.tmpl no longer pins the platform — bin/upgrade_containers.sh regenerates compose from the template, so devices on the new image automatically pick up the Dockerfile's per-board default.

Known pre-existing issue (not introduced here)

view_video in src/anthias_viewer/__init__.py:497-505 has a race: skip_event.clear() → wait(timeout=duration). If something else (e.g. an API state change firing reload) sets skip_event between the clear() and wait(), the very first asset after a viewer restart can be stopped immediately. Not introduced by this PR; worth filing separately.

Checklist

  • I have performed a self-review of my own code.
  • New and existing unit tests pass locally and on CI with my changes.
  • I have done an end-to-end test for Raspberry Pi devices.
  • I have tested my changes for x86 devices.
  • I added a documentation for the changes I have made (when necessary).

…cess

Routes Qt6-board video playback through ``mpv_render_context`` inside
the AnthiasWebview process so a single Qt process holds DRM master /
the Wayland surface for both web pages, images, and video. Fixes the
two-process framebuffer contention that produced 600-2800 vo drops
per 60 s clip on Pi 4 under linuxfb (issue #2904).

- ``src/anthias_webview/`` gains ``VideoView`` (``QOpenGLWidget`` +
  libmpv ``mpv_render_context``) and two new D-Bus slots on
  ``MainWindow`` (``playVideo`` / ``stopVideo``) plus a ``videoEnded``
  signal. ``View`` toggles VideoView visibility alongside the existing
  dual ``QWebEngineView`` / image surface.
- ``src/anthias_viewer/media_player.py``: ``MPVMediaPlayer.play`` /
  ``stop`` switch from ``subprocess.Popen([mpv, ...])`` to
  ``browser_bus.playVideo(uri, options)`` over the same pydbus proxy
  the viewer already uses for ``loadPage`` / ``loadImage``. Options
  dict carries ``hwdec`` / ``audio-device`` / ``video-sync`` /
  ``vd-lavc-threads`` / ``video-rotate`` — the per-board, per-codec
  dispatch and the ALSA / rotation rules from PR #2885 are preserved
  verbatim. ``--vo=...`` and ``--drm-mode=...`` are gone: libmpv-render
  paints into Qt's GL FBO and Qt's eglfs KMS config (Pi 4) pins the
  framebuffer mode now.
- ``docker/Dockerfile.viewer.j2``: ``libmpv-dev`` + ``pkg-config`` in
  the builder, ``libmpv2`` at runtime (dropping the unused ``mpv``
  CLI — ``ffprobe`` ships in the ``ffmpeg`` package). Pi 4 switches
  ``QT_QPA_PLATFORM`` from ``linuxfb`` → ``eglfs`` so Qt has the GL
  context libmpv-render needs; eglfs KMS config pins ``HDMI1``/
  ``HDMI2`` to 1920x1080 and ``QT_SCALE_FACTOR=1`` short-circuits the
  auto-detect that would otherwise pick 2x from the 4K connector EDID.
- ``tests/test_media_player.py``: argv assertions become options-dict
  assertions; ``subprocess.Popen`` mock becomes a ``_browser_bus``
  mock. ``is_playing()`` flips to a local flag (the asset_loop has
  zero callers — only tests).
- ``src/anthias_webview/tests/``: new QtTest unit suite for
  ``VideoView`` (libmpv handle lifecycle, option round-trip via
  ``mpv_get_property_string``, ``--key`` → ``key`` normalisation,
  stop-without-play idempotence, video-rotate value passthrough).
  ``bin/test_webview_cpp.sh`` builds and runs them under
  ``QT_QPA_PLATFORM=offscreen``. Not wired into CI in this PR.
pydbus refuses to coerce a plain Python ``str`` value into a
``GLib.Variant`` when the slot is declared with ``QVariantMap``
(D-Bus ``a{sv}``), surfacing at runtime as
``MPVMediaPlayer.play failed: argument value: Expected GLib.Variant,
but got str`` on every video play. Wrap each option value via
``_marshal_dbus_options`` so the call site hands pydbus a
``dict[str, GLib.Variant]``. Test fixture monkeypatches the helper
to identity so existing options-dict assertions keep working;
a regression test verifies the wrap actually produces ``GLib.Variant``
instances (caught the same way the live error surfaced on the
x86 testbed deploy).
@vpetersson vpetersson requested a review from a team as a code owner May 15, 2026 13:37
@vpetersson vpetersson self-assigned this May 15, 2026
Adds hard-data instrumentation so we can compare the libmpv-embedded
path against PR #2885's 600-2800 vo drops/60 s Pi 4 baseline rather
than inferring "no drops" from low load averages.

VideoView opens ``/data/.anthias/mpv-stats.log`` (bind-mounted from
the host so ``docker exec cat`` reaches it) and writes:

- ``INIT`` at startup (libmpv client API version)
- ``LOADFILE`` on every ``play()`` call — the requested option dict
  (hwdec, audio-device, video-sync, vd-lavc-threads, video-rotate)
- ``FILE_LOADED`` after mpv's decoder probe — ``hwdec-current`` is
  the actual decoder mpv engaged (catches silent SW fallback when
  the requested hwdec didn't whitelist), plus video-codec / w / h /
  container-fps for the asset
- ``SAMPLE`` every 1 s during playback (time-pos, frame-drop-count)
- ``END_FILE`` on clip completion — final drop count + elapsed_ms
- ``STOP`` when MPVMediaPlayer.stop() interrupts mid-play

The 8-case QtTest suite still passes (mpv property round-trip,
option normalisation, stop idempotence, video-rotate passthrough).
The warnings about "/data/.anthias" being missing fire only on the
dev host — production has the bind mount.
…tats file

Diagnostic for HEVC drm-copy silently SW-falling-back under the
render API on Pi 4. Subprocess mpv with --vo=null --hwdec=drm-copy
engages the rpi-hevc-dec block via /dev/video19 + /dev/media0 fine,
but the embedded libmpv path reports hwdec-current= empty for every
HEVC clip while the same 1080p60 H.264 + v4l2m2m-copy engages
cleanly. Verbose log + persisting MPV_EVENT_LOG_MESSAGE rows to
mpv-stats.log lets us see exactly which hwdec init step is failing
without depending on AnthiasWebview's stderr (which sh.Command on
the Python side captures into an unreachable bytes buffer).
QOpenGLWidget renders into an offscreen FBO that Qt's compositor
then blits into the window — two copies per frame on top of
libmpv-render's GL upload. On Pi 4 V3D 6.0 that pushed 1080p60
H.264 to 2973 drops/60 s even with HW decode confirmed engaged
(``[vd] Using hardware decoding (v4l2m2m-copy)``); the bottleneck
was VO throughput, not the decoder. ``QOpenGLWindow`` with
``NoPartialUpdate`` swaps the native window's default framebuffer
directly — same path mpv's own Qt example uses.

VideoView changes from ``QOpenGLWidget : QWidget`` to
``QOpenGLWindow : QPaintDeviceWindow`` (QWindow tree, not widget
tree). ``View`` wraps it with ``QWidget::createWindowContainer``
so MainWindow's existing widget-tree layout (dual QWebEngineView
pair + image canvas + video) stays intact. Visibility / geometry
toggle on the container; the GL render context lives on the inner
QWindow and persists across show/hide so repeated plays don't
re-initialise ``mpv_render_context``.

``QT += opengl`` replaces ``openglwidgets`` in both
``AnthiasWebview.pro`` and ``tests/tests.pro``. 8 QtTest cases
still pass (mpv handle lifecycle, option round-trip, key
normalisation, stop idempotence, video-rotate passthrough), and
the offscreen-platform "QOpenGLWidget not supported" warning that
fired on the dev host is gone — QOpenGLWindow doesn't have that
restriction.
The libmpv-embedded path engaged HW decode correctly on all 4 Qt6
boards but didn't move Pi 4 frame drops below the subprocess-mpv
baseline (562–2973 drops/60 s, same range as PR #2885). Real-device
verbose mpv logging confirmed the decoders engaged; the bottleneck
was V3D 6.0 fillrate through libmpv-render → QOpenGLWidget FBO →
Qt-compositor → eglfs swap. Skipping the FBO indirection by porting
to QOpenGLWindow crashed under eglfs's single-native-window-per-
process limit (reverted at f057198).

QtMultimedia + gstreamer is the next try:

- ``VideoView`` rewritten around QMediaPlayer + QVideoWidget +
  QAudioOutput. QVideoWidget paints inside MainWindow's existing
  eglfs native window, so we don't trip the single-window
  restriction. Stats logger keeps the same /data/.anthias/mpv-stats.log
  schema (INIT / LOADFILE / PLAYING / SAMPLE / END_FILE) with a
  drop estimate computed from container_fps × elapsed − frames-
  delivered (QVideoSink::videoFrameChanged counter).
- ``QT_MEDIA_BACKEND=gstreamer`` is set in the viewer Dockerfile so
  Qt picks the gstreamer backend over its ffmpeg one — the rpi
  ``v4l2slh264dec`` / ``v4l2slh265dec`` elements (in rpt3
  ``gstreamer1.0-plugins-bad``, confirmed via ``dpkg-deb -c`` on
  the .deb) route directly to QVideoSink.
- ``docker/_rpt1-ffmpeg-pin.j2`` extends the rpi-archive pin to
  ``gstreamer1.0-*`` + ``libgstreamer*`` so plugins-bad wins from
  rpt3 over stock Debian (priority bump 100 → 1001).
- ``viewer_extra_apt_dependencies`` swaps libmpv2 for the
  gstreamer1.0-{alsa,libav,plugins-{base,good,bad,ugly}} +
  libqt6multimedia6 + libqt6multimediawidgets6 + qt6-multimedia-dev
  set.
- ``MPVMediaPlayer`` Python options shrink to audio-device +
  video-rotate (Pi 4 only). Removed: hwdec, video-sync,
  vd-lavc-threads, the _PI_HWDEC_BY_CODEC table, _probe_video_codec,
  _pi_hwdec_for_uri. Codec dispatch is now gstreamer's job.
- Python tests drop the ~12 per-codec / ffprobe-dispatch tests;
  C++ tests drop the mpv_get_property_string round-trip in favour
  of QMediaPlayer construction + option-passthrough assertions.
  Codec-gate symmetry test replaced with "gate codecs ⊆
  {h264, hevc}" — broader, catches a relaxation of the upload
  gate at the same time.

Pi 4 perf gain is unverified — Pi 4 testbed went offline
mid-session (along with Pi 5 and Rock Pi 4). Real-device
validation pending.
… env

End-to-end test on x86 caught two bugs:

1. ``player->setSource(QUrl(uri))`` parsed local paths
   (``/data/anthias_assets/...mp4``) as scheme-less relative URLs.
   QMediaPlayer's ffmpeg backend silently refuses to load them — no
   error fired, position stayed at 0 for the whole 12-second
   asset_loop wait. Use ``QUrl::fromLocalFile`` when the URI starts
   with ``/``; absolute URLs (``http://``, ``file://``, ``rtsp://``)
   round-trip through ``QUrl(uri)`` as before.

2. Qt 6.8 dropped the gstreamer multimedia backend upstream (6.5+).
   Debian Trixie ships only ``libffmpegmediaplugin.so`` in
   ``/usr/lib/.../qt6/plugins/multimedia/``. ``QT_MEDIA_BACKEND=
   gstreamer`` silently fell back to ffmpeg, which means our path
   is fundamentally similar to the libmpv-via-render-API attempt
   (ffmpeg decoder → QtMultimedia GL upload). The plan's "zero-copy
   DMA-BUF through gstreamer" claim doesn't hold on this Qt version.

   Pin ``QT_MEDIA_BACKEND=ffmpeg`` explicitly so a future Qt that
   re-introduces the gstreamer backend doesn't silently switch us.
   Keep the ``gstreamer1.0-*`` apt set because the +rpt1 libavcodec
   the ffmpeg backend dlopens IS configured with v4l2_request /
   v4l2m2m hwaccels, and ``gst-launch-1.0`` / ``gst-inspect-1.0``
   stay useful as hardware diagnostics.

Pi 4 perf gain still unverified — same architectural ceiling
applies. The PR continues as architectural cleanup (single Qt
process, no mpv subprocess, smaller option dict) while we measure.
The earlier commit added the full gstreamer1.0-* plugin set under
the assumption QtMultimedia would route through ``v4l2slh*dec``
elements. Qt 6.8 dropped that backend upstream — only
``libffmpegmediaplugin.so`` ships in
``/usr/lib/.../qt6/plugins/multimedia/``. Decode actually goes
through libavcodec directly (the +rpt1 build still carries
``--enable-v4l2-request`` / ``--enable-v4l2-m2m``, so HW decode
still reaches the rpi-hevc-dec + bcm2835-codec hardware), and the
gstreamer packages were dead weight: ~400-500 MB of image bloat
that pushed the Pi 4 viewer image past the SD card's free space
during ``docker load`` (disk hit 0 % before the load completed).

Drop the gstreamer set + the ``QT_MEDIA_BACKEND=ffmpeg`` env
(default selection arrives at ffmpeg anyway, no need to pin). Also
trim ``_rpt1-ffmpeg-pin.j2``'s pin string back to the ffmpeg /
libav* family.
view_video in src/anthias_viewer/__init__.py:495 calls
``view_image('null')`` AFTER ``media_player.play()`` to clear any
leftover image / webpage background while the just-started video
holds the foreground. My loadImage handler was unconditionally
calling ``hideVideoSurface()`` (→ ``videoView->stop()``) on every
loadImage, including the ``'null'`` sentinel, which stopped the
video 66 ms after PLAYING fired.

On Pi 4 this was mostly invisible because the bcm2835-codec
decoder is past its critical init phase at 66 ms; the player
recovered. On Pi 5 the Hantro G2 + CMA allocation for 4K60 HEVC
is still mid-init at 66 ms, the stop() interrupts it, and the
QMediaPlayer leaves position pegged at 0 for the entire 60 s
asset_loop window — black screen, 3600 dropped frames. Subsequent
plays of the same clip work because partial decoder state
persists.

Skip hideVideoSurface when preUri == 'null'. Real image URIs
still tear down the video.
@vpetersson vpetersson changed the title feat(viewer,webview): embed libmpv in AnthiasWebview to eliminate two-process DRM contention feat(viewer,webview): embed QtMultimedia in AnthiasWebview, eliminate two-process DRM contention + Pi 4 drops May 15, 2026
@sonarqubecloud
Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant